External user interface for head worn computing

ABSTRACT

Aspects of the present invention relate to external user interfaces used in connection with head worn computers (HWC).

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/378,191, filed on Apr. 8, 2019, which is a continuation of U.S.application Ser. No. 14/280,752, filed May 19, 2014, now U.S. Pat. No.10,254,856, which is a continuation-in-part of U.S. application Ser. No.14/158,198, filed Jan. 17, 2014, now U.S. Pat. No. 9,939,934, thedisclosures of each which are hereby incorporated by reference in theirentirety.

BACKGROUND Field of the Invention

This invention relates to head worn computing. More particularly, thisinvention relates to external user interfaces related to head worncomputing.

Description of Related Art

Wearable computing systems have been developed and are beginning to becommercialized. Many problems persist in the wearable computing fieldthat need to be resolved to make them meet the demands of the market.

SUMMARY

This Summary introduces certain concepts of head worn computing, and theconcepts are further described below in the Detailed Description and/orshown in the Figures. This Summary should not be considered to describeessential features of the claimed subject matter, nor used to determineor limit the scope of the claimed subject matter.

Aspects of the present invention relate to external user interfaces usedin connection with head worn computers (HWC). Embodiments relate to anexternal user interface that has a physical form intended to be handheld. The hand held user interface may be in the form similar to that ofa writing instrument, such as a pen. In embodiments, the hand held userinterface includes technologies relating to aligning and tracking thealignment of writing patterns on a writing surface through the use ofdisplayed content in a head-worn computer.

These and other systems, methods, objects, features, and advantages ofthe present invention will be apparent to those skilled in the art fromthe following detailed description of the preferred embodiment and thedrawings. All documents mentioned herein are hereby incorporated intheir entirety by reference.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments are described with reference to the following Figures. Thesame numbers may be used throughout to reference like features andcomponents that are shown in the Figures:

FIG. 1 illustrates a head worn computing system in accordance with theprinciples of the present invention.

FIG. 2 illustrates an external user interface in accordance with theprinciples of the present invention.

FIG. 3A to 3C illustrate distance control systems in accordance with theprinciples of the present invention.

FIG. 4A to 4C illustrate force interpretation systems in accordance withthe principles of the present invention.

FIG. 5A to 5C illustrate user interface mode selection systems inaccordance with the principles of the present invention.

FIG. 6 illustrates interaction systems in accordance with the principlesof the present invention.

FIG. 7 illustrates external user interfaces in accordance with theprinciples of the present invention.

FIG. 8 illustrates a pattern recognition system and process inaccordance with the principles of the present invention.

While the invention has been described in connection with certainpreferred embodiments, other embodiments would be understood by one ofordinary skill in the art and are encompassed herein.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT(S)

Aspects of the present invention relate to head-worn computing (“HWC”)systems. HWC involves, in some instances, a system that mimics theappearance of head-worn glasses or sunglasses. The glasses may be afully developed computing platform, such as including computer displayspresented in each of the lenses of the glasses to the eyes of the user.In embodiments, the lenses and displays may be configured to allow aperson wearing the glasses to see the environment through the lenseswhile also seeing, simultaneously, digital imagery, which forms anoverlaid image that is perceived by the person as a digitally augmentedimage of the environment, or augmented reality (“AR”).

HWC involves more than just placing a computing system on a person'shead. The system may need to be designed as a lightweight, compact andfully functional computer display, such as wherein the computer displayincludes a high resolution digital display that provides a high level ofemersion comprised of the displayed digital content and the see-throughview of the environmental surroundings. User interfaces and controlsystems suited to the HWC device may be required that are unlike thoseused for a more conventional computer such as a laptop. For the HWC andassociated systems to be most effective, the glasses may be equippedwith sensors to determine environmental conditions, geographic location,relative positioning to other points of interest, objects identified byimaging and movement by the user or other users in a connected group,and the like. The HWC may then change the mode of operation to match theconditions, location, positioning, movements, and the like, in a methodgenerally referred to as a contextually aware HWC. The glasses also mayneed to be connected, wirelessly or otherwise, to other systems eitherlocally or through a network. Controlling the glasses may be achievedthrough the use of an external device, automatically throughcontextually gathered information, through user gestures captured by theglasses sensors, and the like. Each technique may be further refineddepending on the software application being used in the glasses. Theglasses may further be used to control or coordinate with externaldevices that are associated with the glasses.

Referring to FIG. 1, an overview of the HWC system 100 is presented. Asshown, the HWC system 100 comprises a HWC 102, which in this instance isconfigured as glasses to be worn on the head with sensors such that theHWC 102 is aware of the objects and conditions in the environment 114.In this instance, the HWC 102 also receives and interprets controlinputs such as gestures and movements 116. The HWC 102 may communicatewith external user interfaces 104. The external user interfaces 104 mayprovide a physical user interface to take control instructions from auser of the HWC 102 and the external user interfaces 104 and the HWC 102may communicate bi-directionally to affect the user's command andprovide feedback to the external device 108. The HWC 102 may alsocommunicate bi-directionally with externally controlled or coordinatedlocal devices 108. For example, an external user interface 104 may beused in connection with the HWC 102 to control an externally controlledor coordinated local device 108. The externally controlled orcoordinated local device 108 may provide feedback to the HWC 102 and acustomized GUI may be presented in the HWC 102 based on the type ofdevice or specifically identified device 108. The HWC 102 may alsointeract with remote devices and information sources 112 through anetwork connection 110. Again, the external user interface 104 may beused in connection with the HWC 102 to control or otherwise interactwith any of the remote devices 108 and information sources 112 in asimilar way as when the external user interfaces 104 are used to controlor otherwise interact with the externally controlled or coordinatedlocal devices 108. Similarly, HWC 102 may interpret gestures 116 (e.g.captured from forward, downward, upward, rearward facing sensors such ascamera(s), range finders, IR sensors, etc.) or environmental conditionssensed in the environment 114 to control either local or remote devices108 or 112.

We will now describe each of the main elements depicted on FIG. 1 inmore detail; however, these descriptions are intended to provide generalguidance and should not be construed as limiting. Additional descriptionof each element may also be further described herein.

The HWC 102 is a computing platform intended to be worn on a person'shead. The HWC 102 may take many different forms to fit many differentfunctional requirements. In some situations, the HWC 102 will bedesigned in the form of conventional glasses. The glasses may or may nothave active computer graphics displays. In situations where the HWC 102has integrated computer displays the displays may be configured assee-through displays such that the digital imagery can be overlaid withrespect to the user's view of the environment 114. There are a number ofsee-through optical designs that may be used, including ones that have areflective display (e.g. LCoS, DLP), emissive displays (e.g. OLED, LED),hologram, TIR waveguides, and the like. In addition, the opticalconfiguration may be monocular or binocular. It may also include visioncorrective optical components. In embodiments, the optics may bepackaged as contact lenses. In other embodiments, the HWC 102 may be inthe form of a helmet with a see-through shield, sunglasses, safetyglasses, goggles, a mask, fire helmet with see-through shield, policehelmet with see through shield, military helmet with see-through shield,utility form customized to a certain work task (e.g. inventory control,logistics, repair, maintenance, etc.), and the like.

The HWC 102 may also have a number of integrated computing facilities,such as an integrated processor, integrated power management,communication structures (e.g. cell net, WiFi, Bluetooth, local areaconnections, mesh connections, remote connections (e.g. client server,etc.)), and the like. The HWC 102 may also have a number of positionalawareness sensors, such as GPS, electronic compass, altimeter, tiltsensor, IMU, and the like. It may also have other sensors such as acamera, rangefinder, hyper-spectral camera. Geiger counter, microphone,spectral illumination detector, temperature sensor, chemical sensor,biologic sensor, moisture sensor, ultrasonic sensor, and the like.

The HWC 102 may also have integrated control technologies. Theintegrated control technologies may be contextual based control, passivecontrol, active control, user control, and the like. For example, theHWC 102 may have an integrated sensor (e.g. camera) that captures userhand or body gestures 116 such that the integrated processing system caninterpret the gestures and generate control commands for the HWC 102. Inanother example, the HWC 102 may have sensors that detect movement (e.g.a nod, head shake, and the like) including accelerometers, gyros andother inertial measurements, where the integrated processor mayinterpret the movement and generate a control command in response. TheHWC 102 may also automatically control itself based on measured orperceived environmental conditions. For example, if it is bright in theenvironment the HWC 102 may increase the brightness or contrast of thedisplayed unage. In embodiments, the integrated control technologies maybe mounted on the HWC 102 such that a user can interact with itdirectly. For example, the HWC 102 may have a button(s), touchcapacitive interface, and the like.

As described herein, the HWC 102 may be in communication with externaluser interfaces 104. The external user interfaces may come in manydifferent forms. For example, a cell phone screen may be adapted to takeuser input for control of an aspect of the HWC 102. The external userinterface may be a dedicated UI, such as a keyboard, touch surface,button(s), joy stick, and the like. In embodiments, the externalcontroller may be integrated into another device such as a ring, watch,bike, car, and the like. In each case, the external user interface 104may include sensors (e.g. IMU, accelerometers, compass, altimeter, andthe like) to provide additional input for controlling the HWD 104.

As described herein, the HWC 102 may control or coordinate with otherlocal devices 108. The external devices 108 may be an audio device,visual device, vehicle, cell phone, computer, and the like. Forinstance, the local external device 108 may be another HWC 102, whereinformation may then be exchanged between the separate HWCs 108.

Similar to the way the HWC 102 may control or coordinate with localdevices 106, the HWC 102 may control or coordinate with remote devices112, such as the HWC 102 communicating with the remote devices 112through a network 110. Again, the form of the remote device 112 may havemany forms. Included in these forms is another HWC 102. For example,each HWC 102 may communicate its GPS position such that all the HWCs 102know where all of HWC 102 are located.

Referring to FIG. 2, we now turn to describe a particular external userinterface 104, referred to generally as a pen 200. The pen 200 is aspecially designed external user interface 104 and can operate as a userinterface, such as to many different styles of HWC 102. The pen 200generally follows the form of a conventional pen, which is a familiaruser handled device and creates an intuitive physical interface for manyof the operations to be carried out in the HWC system 100. The pen 200may be one of several user interfaces 104 used in connection withcontrolling operations within the HWC system 100. For example, the HWC102 may watch for and interpret hand gestures 116 as control signals,where the pen 200 may also be used as a user interface with the same HWC102. Similarly, a remote keyboard may be used as an external userinterface 104 in concert with the pen 200. The combination of userinterfaces or the use of just one control system generally depends onthe operation(s) being executed in the HWC's system 100.

While the pen 200 may follow the general form of a conventional pen, itcontains numerous technologies that enable it to function as an externaluser interface 104. FIG. 2 illustrate technologies comprised in the pen200. As can be seen, the pen 200 may include a camera 208, which isarranged to view through lens 202. The camera may then be focused, suchas through lens 202, to image a surface upon which a user is writing ormaking other movements to interact with the HWC 102. There aresituations where the pen 200 will also have an ink, graphite, or othersystem such that what is being written can be seen on the writingsurface. There are other situations where the pen 200 does not have sucha physical writing system so there is no deposit on the writing surface,where the pen would only be communicating data or commands to the HWC102. The lens configuration is described in greater detail herein. Thefunction of the camera is to capture information from an unstructuredwriting surface such that pen strokes can be interpreted as intended bythe user. To assist in the predication of the intended stroke path, thepen 200 may include a sensor, such as an IMU 212. Of course, the IMUcould be included in the pen 200 in its separate parts (e.g. gyro,accelerometer, etc.) or an IMU could be included as a single unit. Inthis instance, the IMU 212 is used to measure and predict the motion ofthe pen 200. In turn, the integrated microprocessor 210 would take theIMU information and camera information as inputs and process theinformation to form a prediction of the pen tip movement.

The pen 200 may also include a pressure monitoring system 204, such asto measure the pressure exerted on the lens 202. As will be described ingreater herein, the pressure measurement can be used to predict theuser's intention for changing the weight of a line, type of a line, typeof brush, click, double click, and the like. In embodiments, thepressure sensor may be constructed using any force or pressuremeasurement sensor located behind the lens 202, including for example, aresistive sensor, a current sensor, a capacitive sensor, a voltagesensor such as a piezoelectric sensor, and the like.

The pen 200 may also include a communications module 218, such as forbi-directional communication with the HWC 102. In embodiments, thecommunications module 218 may be a short distance communication module(e.g. Bluetooth). The communications module 218 may be security matchedto the HWC 102. The communications module 218 may be arranged tocommunicate data and commands to and from the microprocessor 210 of thepen 200. The microprocessor 210 may be programmed to interpret datagenerated from the camera 208, IMU 212, and pressure sensor 204, and thelike, and then pass a command onto the HWC 102 through thecommunications module 218, for example. In another embodiment, the datacollected from any of the input sources (e.g. camera 108, IMU 212,pressure sensor 104) by the microprocessor may be communicated by thecommunication module 218 to the HWC 102, and the HWC 102 may performdata processing and prediction of the user's intention when using thepen 200. In yet another embodiment, the data may be further passed onthrough a network 110 to a remote device 112, such as a server, for thedata processing and prediction. The commands may then be communicatedback to the HWC 102 for execution (e.g. display writing in the glassesdisplay, make a selection within the UI of the glasses display, controla remote external device 112, control a local external device 108), andthe like. The pen may also include memory 214 for long or short termuses.

The pen 200 may also include a number of physical user interfaces, suchas quick launch buttons 222, a touch sensor 220, and the like. The quicklaunch buttons 222 may be adapted to provide the user with a fast way ofjumping to a software application in the HWC system 100. For example,the user may be a frequent user of communication software packages (e.g.email, text, Twitter, Instagram, Facebook, Google+, and the like), andthe user may program a quick launch button 222 to command the HWC 102 tolaunch an application. The pen 200 may be provided with several quicklaunch buttons 222, which may be user programmable or factoryprogrammable. The quick launch button 222 may be programmed to performan operation. For example, one of the buttons may be programmed to clearthe digital display of the HWC 102. This would create a fast way for theuser to clear the screens on the HWC 102 for any reason, such as forexample to better view the environment. The quick launch buttonfunctionality will be discussed in further detail below. The touchsensor 220 may be used to take gesture style input from the user. Forexample, the user may be able to take a single finger and run it acrossthe touch sensor 220 to affect a page scroll.

The pen 200 may also include a laser pointer 224. The laser pointer 224may be coordinated with the IMU 212 to coordinate gestures and laserpointing. For example, a user may use the laser 224 in a presentation tohelp with guiding the audience with the interpretation of graphics andthe IMU 212 may, either simultaneously or when the laser 224 is off,interpret the user's gestures as commands or data input.

FIGS. 3A-C illustrate several embodiments of lens and cameraarrangements 300 for the pen 200. One aspect relates to maintaining aconstant distance between the camera and the writing surface to enablethe writing surface to be kept in focus for better tracking of movementsof the pen 200 over the writing surface. Another aspect relates tomaintaining an angled surface following the circumference of the writingtip of the pen 200 such that the pen 200 can be rolled or partiallyrolled in the user's hand to create the feel and freedom of aconventional writing instrument.

FIG. 3A illustrates an embodiment of the writing lens end of the pen200. The configuration includes a ball lens 304, a camera or imagecapture surface 302, and a domed cover lens 308. In this arrangement,the camera views the writing surface through the ball lens 304 and domecover lens 308. The ball lens 304 causes the camera to focus such thatthe camera views the writing surface when the pen 200 is held in thehand in a natural writing position, such as with the pen 200 in contactwith a writing surface. In embodiments, the ball lens 304 should beseparated from the writing surface to obtain the highest resolution ofthe writing surface at the camera 302. In embodiments, the ball lens 304is separated by approximately 1 to 3 mm. In this configuration, thedomed cover lens 308 provides a surface that can keep the ball lens 304separated from the writing surface at a constant distance, such assubstantially independent of the angle used to write on the writingsurface. For instance, in embodiments the field of view of the camera inthis arrangement would be approximately 60 degrees.

The domed cover lens, or other lens 308 used to physically interact withthe writing surface, will be transparent or transmissive within theactive bandwidth of the camera 302. In embodiments, the domed cover lens308 may be spherical or other shape and comprised of glass, plastic,sapphire, diamond, and the like. In other embodiments where lowresolution imaging of the surface is acceptable. The pen 200 can omitthe domed cover lens 308 and the ball lens 304 can be in direct contactwith the surface.

FIG. 3B illustrates another structure where the construction is somewhatsimilar to that described in connection with FIG. 3A; however thisembodiment does not use a dome cover lens 308, but instead uses a spacer310 to maintain a predictable distance between the ball lens 304 and thewriting surface, wherein the spacer may be spherical, cylindrical,tubular or other shape that provides spacing while allowing for an imageto be obtained by the camera 302 through the lens 304. In a preferredembodiment, the spacer 310 is transparent. In addition, while the spacer310 is shown as spherical, other shapes such a an oval, doughnut shape,half sphere, cone, cylinder or other form may be used.

FIG. 3C illustrates yet another embodiment, where the structure includesa post 314, such as running through the center of the lensed end of thepen 200. The post 314 may be an ink deposition system (e.g. inkcartridge), graphite deposition system (e.g. graphite holder), or adummy post whose purpose is mainly only that of alignment. The selectionof the post type is dependent on the pen's use. For instance, in theevent the user wants to use the pen 200 as a conventional ink depositingpen as well as a fully functional external user interface 104, the inksystem post would be the best selection. If there is no need for the‘writing’ to be visible on the writing surface, the selection would bethe dummy post. The embodiment of FIG. 3C includes camera(s) 302 and anassociated lens 312, where the camera 302 and lens 312 are positioned tocapture the writing surface without substantial interference from thepost 314. In embodiments, the pen 200 may include multiple cameras 302and lenses 312 such that more or all of the circumference of the tip 314can be used as an input system. In an embodiment, the pen 200 includes acontoured grip that keeps the pen aligned in the user's hand so that thecamera 302 and lens 312 remains pointed at the surface.

Another aspect of the pen 200 relates to sensing the force applied bythe user to the writing surface with the pen 200. The force measurementmay be used in a number of ways. For example, the force measurement maybe used as a discrete value, or discontinuous event tracking, andcompared against a threshold in a process to determine a user's intent.The user may want the force interpreted as a ‘click’ in the selection ofan object, for instance. The user may intend multiple force exertionsinterpreted as multiple clicks. There may be times when the user holdsthe pen 200 in a certain position or holds a certain portion of the pen200 (e.g. a button or touch pad) while clicking to affect a certainoperation (e.g. a ‘right click’). In embodiments, the force measurementmay be used to track force and force trends. The force trends may betracked and compared to threshold limits, for example. There may be onesuch threshold limit, multiple limits, groups of related limits, and thelike. For example, when the force measurement indicates a fairlyconstant force that generally falls within a range of related thresholdvalues, the microprocessor 210 may interpret the force trend as anindication that the user desires to maintain the current writing style,writing tip type, line weight, brush type, and the like. In the eventthat the force trend appears to have gone outside of a set of thresholdvalues intentionally, the microprocessor may interpret the action as anindication that the user wants to change the current writing style,writing tip type, line weight, brush type, and the like. Once themicroprocessor has made a determination of the user's intent, a changein the current writing style, writing tip type, line weight, brush type,and the like, may be executed. In embodiments, the change may be notedto the user (e.g. in a display of the HWC 102), and the user may bepresented with an opportunity to accept the change.

FIG. 4A illustrates an embodiment of a force sensing surface tip 400 ofa pen 200. The force sensing surface tip 400 comprises a surfaceconnection tip 402 (e.g. a lens as described herein elsewhere) inconnection with a force or pressure monitoring system 204. As a useruses the pen 200 to write on a surface or simulate writing on a surfacethe force monitoring system 204 measures the force or pressure the userapplies to the writing surface and the force monitoring systemcommunicates data to the microprocessor 210 for processing. In thisconfiguration, the microprocessor 210 receives force data from the forcemonitoring system 204 and processes the data to make predictions of theuser's intent in applying the particular force that is currently beingapplied. In embodiments, the processing may be provided at a locationother than on the pen (e.g. at a server in the HWC system 100, on theHWC 102). For clarity, when reference is made herein to processinginformation on the microprocessor 210, the processing of informationcontemplates processing the information at a location other than on thepen. The microprocessor 210 may be programmed with force threshold(s),force signature(s), force signature library and/or other characteristicsintended to guide an inference program in determining the user'sintentions based on the measured force or pressure. The microprocessor210 may be further programmed to make inferences from the forcemeasurements as to whether the user has attempted to initiate a discreteaction (e.g. a user interface selection ‘click’) or is performing aconstant action (e.g. writing within a particular writing style). Theinferencing process is important as it causes the pen 200 to act as anintuitive external user interface 104.

FIG. 4B illustrates a force 408 versus time 410 trend chart with asingle threshold 418. The threshold 418 may be set at a level thatindicates a discrete force exertion indicative of a user's desire tocause an action (e.g. select an object in a GUI). Event 412, forexample, may be interpreted as a click or selection command because theforce quickly increased from below the threshold 418 to above thethreshold 418. The event 414 may be interpreted as a double clickbecause the force quickly increased above the threshold 418, decreasedbelow the threshold 418 and then essentially repeated quickly. The usermay also cause the force to go above the threshold 418 and hold for aperiod indicating that the user is intending to select an object in theGUI (e.g. a GUI presented in the display of the HWC 102) and ‘hold’ fora further operation (e.g. moving the object).

While a threshold value may be used to assist in the interpretation ofthe user's intention, a signature force event trend may also be used.The threshold and signature may be used in combination or either methodmay be used alone. For example, a single-click signature may berepresented by a certain force trend signature or set of signatures. Thesingle-click signature(s) may require that the trend meet a criteria ofa rise time between x any y values, a hold time of between a and bvalues and a fall time of between c and d values, for example.Signatures may be stored for a variety of functions such as click,double click, right click, hold, move, etc. The microprocessor 210 maycompare the real-time force or pressure tracking against the signaturesfrom a signature library to make a decision and issue a command to thesoftware application executing in the GUI.

FIG. 4C illustrates a force 408 versus time 410 trend chart withmultiple thresholds 418. By way of example, the force trend is plottedon the chart with several pen force or pressure events. As noted, thereare both presumably intentional events 420 and presumablynon-intentional events 422. The two thresholds 418 of FIG. 4C createthree zones of force: a lower, middle and higher range. The beginning ofthe trend indicates that the user is placing a lower zone amount offorce. This may mean that the user is writing with a given line weightand does not intend to change the weight, the user is writing. Then thetrend shows a significant increase 420 in force into the middle forcerange. This force change appears, from the trend to have been sudden andthereafter it is sustained. The microprocessor 210 may interpret this asan intentional change and as a result change the operation in accordancewith preset rules (e.g. change line width, increase line weight, etc.).The trend then continues with a second apparently intentional event 420into the higher-force range. During the performance in the higher-forcerange, the force dips below the upper threshold 418. This may indicatean unintentional force change and the microprocessor may detect thechange in range however not affect a change in the operations beingcoordinated by the pen 200. As indicated above, the trend analysis maybe done with thresholds and/or signatures.

Generally, in the present disclosure, instrument stroke parameterchanges may be referred to as a change in line type, line weight, tiptype, brush type, brush width, brush pressure, color, and other forms ofwriting, coloring, painting, and the like.

Another aspect of the pen 200 relates to selecting an operating mode forthe pen 200 dependent on contextual information and/or selectioninterface(s). The pen 200 may have several operating modes. Forinstance, the pen 200 may have a writing mode where the userinterface(s) of the pen 200 (e.g. the writing surface end, quick launchbuttons 222, touch sensor 220, motion based gesture, and the like) isoptimized or selected for tasks associated with writing. As anotherexample, the pen 200 may have a wand mode where the user interface(s) ofthe pen is optimized or selected for tasks associated with software ordevice control (e.g. the HWC 102, external local device, remote device112, and the like). The pen 200, by way of another example, may have apresentation mode where the user interface(s) is optimized or selectedto assist a user with giving a presentation (e.g. pointing with thelaser pointer 224 while using the button(s) 222 and/or gestures tocontrol the presentation or applications relating to the presentation).The pen may, for example, have a mode that is optimized or selected fora particular device that a user is attempting to control. The pen 200may have a number of other modes and an aspect of the present inventionrelates to selecting such modes.

FIG. 5A illustrates an automatic user interface(s) mode selection basedon contextual information. The microprocessor 210 may be programmed withIMU thresholds 514 and 512. The thresholds 514 and 512 may be used asindications of upper and lower bounds of an angle 504 and 502 of the pen200 for certain expected positions during certain predicted modes. Whenthe microprocessor 210 determines that the pen 200 is being held orotherwise positioned within angles 502 corresponding to writingthresholds 514, for example, the microprocessor 210 may then institute awriting mode for the pen's user interfaces. Similarly, if themicroprocessor 210 determines (e.g. through the IMU 212) that the pen isbeing held at an angle 504 that falls between the predetermined wandthresholds 512, the microprocessor may institute a wand mode for thepen's user interface. Both of these examples may be referred to ascontext based user interface mode selection as the mode selection isbased on contextual information (e.g. position) collected automaticallyand then used through an automatic evaluation process to automaticallyselect the pen's user interface(s) mode.

As with other examples presented herein, the microprocessor 210 maymonitor the contextual trend (e.g. the angle of the pen over time) in aneffort to decide whether to stay in a mode or change modes. For example,through signatures, thresholds, trend analysis, and the like, themicroprocessor may determine that a change is an unintentional changeand therefore no user interface mode change is desired.

FIG. 5B illustrates an automatic user interface(s) mode selection basedon contextual information. In this example, the pen 200 is monitoring(e.g. through its microprocessor) whether or not the camera at thewriting surface end 208 is imaging a writing surface in close proximityto the writing surface end of the pen 200. If the pen 200 determinesthat a writing surface is within a predetermined relatively shortdistance, the pen 200 may decide that a writing surface is present 502and the pen may go into a writing mode user interface(s) mode. In theevent that the pen 200 does not detect a relatively close writingsurface 504, the pen may predict that the pen is not currently beingused to as a writing instrument and the pen may go into a non-writinguser interface(s) mode.

FIG. 5C illustrates a manual user interface(s) mode selection. The userinterface(s) mode may be selected based on a twist of a section 508 ofthe pen 200 housing, clicking an end button 510, pressing a quick launchbutton 222, interacting with touch sensor 220, detecting a predeterminedaction at the pressure monitoring system (e.g. a click), detecting agesture (e.g. detected by the IMU), etc. The manual mode selection mayinvolve selecting an item in a GUI associated with the pen 200 (e.g. animage presented in the display of HWC 102).

In embodiments, a confirmation selection may be presented to the user inthe event a mode is going to change. The presentation may be physical(e.g. a vibration in the pen 200), through a GUI, through a lightindicator, etc.

FIG. 6 illustrates a couple pen use-scenarios 600 and 601. There aremany use scenarios and we have presented a couple in connection withFIG. 6 as a way of illustrating use scenarios to further theunderstanding of the reader. As such, the use-scenarios should beconsidered illustrative and non-limiting.

Use scenario 600 is a writing scenario where the pen 200 is used as awriting instrument. In this example, quick launch button 122A is pressedto launch a note application 610 in the GUI 608 of the HWC 102 display604. Once the quick launch button 122A is pressed, the HWC 102 launchesthe note program 610 and puts the pen into a writing mode. The user usesthe pen 200 to scribe symbols 602 on a writing surface, the pen recordsthe scribing and transmits the scribing to the HWC 102 where symbolsrepresenting the scribing are displayed 612 within the note application610.

Use scenario 601 is a gesture scenario where the pen 200 is used as agesture capture and command device. In this example, the quick launchbutton 122B is activated and the pen 200 activates a wand mode such thatan application launched on the HWC 102 can be controlled. Here, the usersees an application chooser 618 in the display(s) of the HWC 102 wheredifferent software applications can be chosen by the user. The usergestures (e.g. swipes, spins, turns, etc.) with the pen to cause theapplication chooser 618 to move from application to application. Oncethe correct application is identified (e.g. highlighted) in the chooser618, the user may gesture or click or otherwise interact with the pen200 such that the identified application is selected and launched. Oncean application is launched, the wand mode may be used to scroll, rotate,change applications, select items, initiate processes, and the like, forexample.

In an embodiment, the quick launch button 122A may be activated and theHWC 102 may launch an application chooser presenting to the user a setof applications. For example, the quick launch button may launch achooser to show all communication programs (e.g. SMS, Twitter,Instagram, Facebook, email, etc.) available for selection such that theuser can select the program the user wants and then go into a writingmode. By way of further example, the launcher may bring up selectionsfor various other groups that are related or categorized as generallybeing selected at a given time (e.g. Microsoft Office products,communication products, productivity products, note products,organizational products, and the like)

FIG. 7 illustrates yet another embodiment of the present invention. FIG.700 illustrates a watchband clip on controller 700. The watchband clipon controller may be a controller used to control the HWC 102 or devicesin the HWC system 100. The watchband clip on controller 700 has afastener 718 (e.g. rotatable clip) that is mechanically adapted toattach to a watchband, as illustrated at 704.

The watchband controller 700 may have quick launch interfaces 708 (e.g.to launch applications and choosers as described herein), a touch pad714 (e.g. to be used as a touch style mouse for GIB control in a HWC 102display) and a display 712. The clip 718 may be adapted to fit a widerange of watchbands so it can be used in connection with a watch that isindependently selected for its function. The clip, in embodiments, isrotatable such that a user can position it in a desirable manner. Inembodiments the clip may be a flexible strap. In embodiments, theflexible strap may be adapted to be stretched to attach to a hand,wrist, finger, device, weapon, and the like.

In embodiments, the watchband controller may be configured as aremovable and replaceable watchband. For example, the controller may beincorporated into a band with a certain width, segment spacing's, etc.such that the watchband, with its incorporated controller, can beattached to a watch body. The attachment, in embodiments, may bemechanically adapted to attach with a pin upon which the watchbandrotates. In embodiments, the watchband controller may be electricallyconnected to the watch and/or watch body such that the watch, watch bodyand/or the watchband controller can communicate data between them.

The watchband controller may have 3-axis motion monitoring (e.g. throughan IMU, accelerometers, magnetometers, gyroscopes, etc.) to capture usermotion. The user motion may then be interpreted for gesture control.

In embodiments, the watchband controller may comprise fitness sensorsand a fitness computer. The sensors may track heart rate, caloriesburned, strides, distance covered, and the like. The data may then becompared against performance goals and/or standards for user feedback.

Another aspect of the present invention relates to tracking penmovements with the assistance of a camera and displayed content in a HWC102. In embodiments, content is presented in a see-through display of ahead-worn computer to provide a virtual guide for the wearer who wantsto make motions with a pen, finger, or other interface and have themotions interpreted for pattern recognition. As described in connectionwith pen embodiments disclosed herein elsewhere, an IMU or pen-tipcamera may be used to monitor the motion of a pen in order to predictwhat patterns are being drawn. The IMU and/or pen tip camera may sufferfrom electronic or optical drift and the drift may cause inaccuracies inthe pattern prediction. In embodiments, to augment the IMU and/or pentip camera motion predictions the virtual guide is provided tocompensate for the drift. The pen motions may be captured by a cameraon-board the HWC 102 while the wearer is writing with the guidance ofthe virtual line. Knowing that the wearer is using the virtual line as aguide, the relative position between the pen tip and virtual line can beused to reduce or eliminate drift issues.

In embodiments, digital content is presented to a wearer of the HWC 102and the wearer moves the pen 200 along a writing surface guided by thedigital content for pattern recordation, recognition and presentationassistance. In embodiments, a camera in the HWC 102 images and tracksthe positions of the pen 200 for pattern recordation and recognitionassistance. In embodiments, both the digital content and the cameracapturing the pen positions are used for pattern recordation andrecognition assistance. In embodiments, the digital content, cameracapture, in-pen camera capture, in-pen IMU, etc. may be used incombination for pattern recordation and recognition assistance. Inembodiments, the relative positions of the pen strokes to the virtualline may be presented in the HWC 102 displays in relation to the virtualline. For example, the wearer of the HWC 102 may be scribing without inkin relation to the virtual line that he perceives and as presented inthe HWC 102 display, the on-board HWC 102 camera may capture thescribing, a processor may interpret the imaged scribing in relation tothe line such that the scribing can be converted into digital content tobe displayed in the HWC 102 display in relation to the virtual line.

FIG. 8 illustrates a system where a camera in the HWC 102 is used totrack pen 200 motions and digital content is presented to the wearer ofthe HWC 102 to assist the wearer with writing within a structure. Inthis embodiment, digital content in the form of a line 804 is presentedin an FOY 802 of the HWC 102. The wearer can see through the FOY 802 sothe line 804 appears to augment the surrounding environment's view forthe wearer. The line may be ‘fixed’ to a spot in the environment suchthat when the wearer turns his head and hence changes the position ofthe HWC 102, the line appears to stay in position with respect to theenvironment. In embodiments, the camera in the HWC 102 may image theenvironment and track the relative movement of the HWC 102 with respectto the environment such that the line 804 can be positioned and movedwithin the FOY in accordance with the imaged movements to maintainvisual alignment of the line with a point, object, marker, etc. in theenvironment. This configuration presents a virtual line in theenvironment that does not appear to move as the wearer's head moves. Thevirtual line can provide the wearer with guidance on where to make penstrokes. The line can be thought of as a line on a piece of paper so thewearer can write, or make strokes in a writing pattern, along thevirtual line to make prediction of the lines pattern more accurate andovercome drift errors that may otherwise be apparent when attempting torecord the movements and predict the patterns.

With the virtual line presented and virtually connected to a position inthe environment, the wearer can use the line for guidance when makingwriting patterns. The HWC 102 camera can also be used to track themovements of the pen 200 relative to the position of the virtual line.This may be used to better predict the patterns indicated by thewearer's pen strokes. As described herein elsewhere, the pen 200 maytrack its motions through a pen tip camera and IMU. In embodiments, thepen tip camera and IMU may track the pen's motion and the camera may beused to track the motion of the pen relative to the virtual line. Eachof these inputs may be used to track, record and predict what it beingwritten.

In embodiments, the camera in the HWC 102 captures images of thewearer's pen's motion while the wearer is using the pen to make patternswith the virtual line as a guide. The virtual line may then be overlaidon the captured images of the motion to assist with the patternanalysis. In embodiments, once the overlay is made, one can see oranalyze how the pen pattern moved with respect to the position of thevirtual line as the wearer may be viewed the virtual line. The patternanalysis may involve interpreting the IMU motion detection, in-penmotion detection, and/or the pen's motion as captured through the HWC102 camera relative to the virtual line. For example, if the IMUindicates that the pen shifted away from the wearer but the position ofthe pen relative to the virtual line indicates the pen was not moving,the portion of IMU data that indicated the shift may be discounted inthe prediction analysis. The virtual line pattern analysis may be donein real-time, after the fact, etc. The pattern recognition may be doneon a processor on-board the HWC 102, remote from the HWC 102, orpartially on-board and remotely.

In embodiments, the virtual line may take any number of forms. Forexample, the virtual line may be a line, part of a virtual note, part ofa virtual message template, etc. The line may also change positions andshapes depending on the wearer's needs. For example, the wearer may wantto trace a pattern that is being displayed as digital content and thedigital content may be presented as a consolidated image, part of animage, image in a line-by-line presentation format, etc. In embodiments,this system may be used for lessons on writing, painting, drawing, etc.

Although embodiments of HWC have been described in language specific tofeatures, systems, computer processes and/or methods, the appendedclaims are not necessarily limited to the specific features, systems,computer processes and/or methods described. Rather, the specificfeatures, systems, computer processes and/or and methods are disclosedas non-limited example implementations of HWC. All documents referencedherein are hereby incorporated by reference.

We claim:
 1. A method, comprising: generating a virtual line as digitalcontent in a see-through display of a wearable head device; presentingthe virtual line as a guide for alignment of pen strokes written by awearer of the wearable head device; tracking a relative motion of a penheld by the wearer as the pen is moving along a surface when guided bythe virtual line, wherein the tracking comprises: detecting, via an IMUlocated on the pen, movement of the pen; and capturing images of the penas the pen moves relative to the virtual line, using a camera in thewearable head device; interpreting the relative motion of the pen todetermine a written pattern, wherein interpreting the relative motion ofthe pen includes: determining, based on the IMU, a first relative motionbetween a tip of the pen and the virtual line; determining, based on thecaptured images, a second relative motion between a tip of the pen andthe virtual line; and in accordance with a determination of adiscrepancy between the first relative motion and the second relativemotion, interpreting the relative motion of the pen based on the secondrelative motion.
 2. The method of claim 1, wherein the virtual line ispresented as a portion of a note.
 3. The method of claim 1, wherein thevirtual line is presented as a portion of a message template.
 4. Themethod of claim 1, wherein the virtual line is presented as a portion ofan email template.
 5. The method of claim 1, wherein the virtual line ispresented as a portion of a writing teaching platform.
 6. The method ofclaim 1, wherein the virtual line is presented as a portion of apainting teaching platform.
 7. The method of claim 1 wherein the virtualline is presented as a portion of a drawing teaching platform.
 8. Themethod of claim 1, wherein the written pattern is presented in thesee-through display relative to the virtual line.
 9. The method of claim8, wherein the presentation is in real time.
 10. The method of claim 1,wherein interpreting the relative motion of the pen includes determininga relative position between the tip of the pen and the virtual line. 11.The method of claim 1, wherein tracking a relative motion of a pen heldby the wearer as the pen is moving along a surface when guided by thevirtual line further comprises capturing surface images using a pen tipcamera located on the pen.
 12. The method of claim 11, whereininterpreting the relative motion of the pen to determine a writtenpattern further comprises determining, based on the surface images, athird relative motion between a tip of the pen and the virtual line andin accordance with the discrepancy between the first relative motion andthe second relative motion, interpreting the relative motion of the penfurther based on the third relative motion.
 13. A non-transitorycomputer-readable storage medium storing one or more programs, the oneor more programs comprising instructions, which when executed by one ormore processors of one or more electronic devices, cause the electronicdevices to: generate a virtual line as digital content in a see-throughdisplay of a wearable head device; present the virtual line as a guidefor alignment of pen strokes written by a wearer of the wearable headdevice; track a relative motion of a pen held by the wearer as the penis moving along a surface when guided by the virtual line, wherein thetracking comprises: detecting, via an IMU located on the pen, movementof the pen; and capturing images of the pen as the pen moves relative tothe virtual line, using a camera in the wearable head device; interpretthe relative motion of the pen to determine a written pattern, whereininterpreting the relative motion of the pen includes: determining, basedon the IMU, a first relative motion between a tip of the pen and thevirtual line; determining, based on the captured images, a secondrelative motion between a tip of the pen and the virtual line; and inaccordance with a determination of a discrepancy between the firstrelative motion and the second relative motion, interpreting therelative motion of the pen based on the second relative motion.
 14. Thenon-transitory computer-readable storage medium of claim 13, whereininterpreting the relative motion of the pen includes determining arelative position between the tip of the pen and the virtual line. 15.The non-transitory computer-readable storage medium of claim 13, whereintracking a relative motion of a pen held by the wearer as the pen ismoving along a surface when guided by the virtual line further comprisescapturing surface images using a pen tip camera located on the pen. 16.The non-transitory computer-readable storage medium of claim 15, whereininterpreting the relative motion of the pen to determine a writtenpattern further comprises determining, based on the surface images, athird relative motion between a tip of the pen and the virtual line andin accordance with the discrepancy between the first relative motion andthe second relative motion, interpreting the relative motion of the penfurther based on the third relative motion.
 17. The non-transitorycomputer-readable storage medium of claim 13, wherein the writtenpattern is presented in the see-through display relative to the virtualline.
 18. The non-transitory computer-readable storage medium of claim17, wherein the presentation is in real time.
 19. The non-transitorycomputer-readable storage medium of claim 13, wherein the virtual lineis presented as a portion of a note.
 20. The non-transitorycomputer-readable storage medium of claim 13, wherein the virtual lineis presented as a portion of a message template.